Image Processing 2

Table of Contents

This is a “pair” assignment, which means that if you are working on a team with someone else, you and your partner should do your best to engage in the pair programming model. At any point in time, one of you is “driving,” i.e. actually using the keyboard and mouse. The other is actively engaged following along, preventing bugs, and providing ideas.

You should make sure that over the course of an assignment that you spend roughly the same amount of time each “driving.” I will also ask you to turn in a form rating the work that your partner does. My recommendation is to take turns approximately every 15 minutes or so. Set a timer to help you remember.

1. Overview

Movies and video conferencing often use a technique called chroma key. With chroma key, the actors in a scene are filmed as they perform in front of a solid color screen (perhaps green, or blue). Later on, after the scene has been filmed, the special effects people remove that color from the scene and replace it with the actual scene: a dinosaur, the New York City skyline, outer space, etc. This technique is also regularly used by TV weather forecasters to make it appear as though they are standing in front of a map.

In this assignment, you will be using the same technique.

2. Get started

Create a folder in which to store your work for this assignment.

  • If you are working on your own computer, it’s up to you where to put the folder. Your desktop is likely as good a place as any. Make a folder titled image2.
  • If you are working in the labs in Olin, make sure to first mount the COURSES folder, so that you won’t lose your code when you log out. Once you’ve done so, open up Finder, then navigate to your personal student work folder. You can then make a image2 folder within there.
  • Once you’ve done so, you should then open up your new folder in VS Code. To do so, start up VS Code, then drag your folder onto the VS Code window. This should open up the folder within VS Code. If you are asked, click that you trust the authors.

To facilitate reading in, storing, and manipulating image files in “standard” formats (like .jpg, .gif, and .png), we will again use images.py, which is included in your repl. Here is the documentation for images.py.

You’ll also need some images to work with. First, you’ll need an image of something in front of a green screen or other solid background. The following links are for images I found online that had Creative Commons licenses that permitted me to resize them and redistribute them for the purpose at hand. I’ve included links to the original images and their licenses. You’re welcome to use your own images for this assignment if you like, but you need to have permission to use them. That means that you’ve taken them yourself, or you’ve included a link or other text indicating a license for the image granting you to use it.

You’ll also need some background images. The following images were all taken by Prof. Amy Csizmar Dalal, and she has granted permission for their use.

3. Your tasks

The background pictures depict some sort of scenery, while the first set of images portray a subject appearing against a plain background. We can combine these images into a single image by replacing the green pixels in one of the original pictures with pixels from one of the scenery pictures. To do this, we have to figure out which pixels correspond to the background (and can be changed) and which ones correspond to the subject (and should be left alone).

In a Python file called photolab2.py, create the following functions:

3.1. Replacing the background with a solid color

Your first function should be defined as: 

replaceBackgroundWithColor(image, backgroundColor, radius, replacementColor)

This function should create and return a new image that replaces all background pixels in the original image by the color specified by the supplied replacement red, green, and blue values. The background red, green, and blue values are used to indicate what is considered “green” for the background. Each image I have provided has a slightly different shade of green, so the background color needs to be specified along with the image.

Note that backgroundColor and replacementColor are expected to be Python tuples. Here is an example of how I might call the above code in a main function: 

origImage = FileImage('somefile.jpg')
greenBackground = Pixel(86, 164, 104)
purpleBackground = Pixel(204, 51, 255)
radius = 20
newImage = replaceBackgroundWithColor(origImage, greenBackground, radius, purpleBackground)

Finally, you should take into account the fact that the background color isn’t perfectly uniform. You’ll need to use some kind of range around it. The radius parameter indicates the size of that range. For example, if backgroundColor is (86, 164, 104), you should replace any pixels that satisfy all three of the following:

  • red values in the range from 86-radius to 86+radius, inclusive
  • green values in the range from 164-radius to 164+radius, inclusive
  • blue values in the range from 104-radius to 104+radius, inclusive

3.2. Replacing the background with an image

Create a Python function defined as: 

replaceBackgroundWithImage(originalImage, backgroundColor, radius, replacementImage)

This function should create and return a new image that replaces all background pixels in the original image with pixels taken from the replacement image. Note that the original image and the replacement image need to have the same dimensions for this technique to produce the desired effect. It should have the same behavior as the previous function regarding ranges of values.

Note that while working on replaceBackgroundWithColor is likely to be conceptually helpful for you in working on replaceBackgroundWithImage, these should be two separate functions. One does not depend on or require the other.

4. Test your code

I have also included testing file tests_m.py, which we’ll use to test if you’re doing your code correctly. You can use pytest, as we’ve done in previous assignments, to test your code yourself.

5. Exemplary

If you complete the above successfully (and have good style in your code), you will receive nearly all the points for the assignment. If you get this far, you should feel proud of your achievements! If you want to push yourself harder and go for the exemplary grade, do the following extra challenge.

Zoom offers functionality to do this without needing an actual green screen; it automatically detects which pixels are your head and which ones are part of a background – sometimes with hilarious results. Zoom is likely using some much more interesting AI algorithms to pull this off, but you can take a first stab at it.

For this part, create another function defined as: 

smartReplaceBackgroundWithImage(originalImage1, originalImage2, replacementImage)

Notice that this function doesn’t take a parameter indicating what the background color is.

If you want to do this exercise, take two pictures of yourself in front of a non-homogenous background (i.e., your usual Zooming location). Move your head around between the pictures so that you’ve got your head in at least somewhat different locations between the two photos. Then write code to see if you can detect which pixels are the background pixels, and which ones are not, based on how much difference there is between the two images. Your function should return the first image, but with the background updated with the replacement image.

There’s no automated test code for this portion: make it clear in your submission with a README file what you’ve done, and how the graders can see if you’ve done it. You don’t need to achieve perfection: show us that you’ve made solid progress at achieving this. But the important thing is to make sure you’re doing so by comparing the two original images, and not by trying to hardwire into the code what color ranges represent the background.

6. Grading

You will receive an M for this assignment if…

  • when we run pytest tests_m.py, your program passes the tests.
  • your program is written to work in general, and not to only work for the specific tests that we have provided.
  • when we run python photolab2.py, it shows us a demonstration of the functions working on at least one of the test images that we have provided.

You will receive a grade of E for this assignment if you satisfy the above M requirements, and …

  • your programs have a comment at the top of each function with at least 5 words describing what the function does.
  • every one of your variable names is meaningful in some way. (Names such as thing, number, etc. are not meaningful.)
  • You have at least one other comment near what you think is the trickiest part of your code, describing how it works
  • your code demonstrates a clear sequence of actions to achieve the goal at hand, and each piece is essential. Your code does not have notably more cases or conditions than it needs to.
  • you have successfully achieved the criteria described in the exemplary section above, particularly regarding the last paragraph there.

7. Submit your work

When finished, zip up your code and submit your work through Moodle.

Good luck, and have fun! Remember that lab assistants are available to help out if you need it, and you can attend prefect sessions as well.

Author: Dave Musicant